Using jsPsych and cognition.run

Session 7 - JATOS


Translations available

Disclaimer: may not be very accurate…


Worksheet overview

Aims

By the end of this worksheet you should be able to:

  • program your own experiments in jsPsych
  • host the experiment online using cognition.run
  • use the participant data for analysis
  • apply the basic skills you have learnt for your own purposes
  • learn some extra skills such as HTML, javascript, CSS and JSON

Pre-requisites

To complete the aims you will need to:

  • follow this worksheet
  • ask questions if you are not sure/be able to google
  • have a working computer and internet connection
  • be patient when things do not work

You do not need to:

  • have any programming knowledge
  • have high computer literacy
  • know anything about jsPsych, cognition.run, html, css or javascript
  • be a linguist

Structure

The worksheet will go through the following sections:

- working with JATOS
- working with piloting on a local server
- working with the FF server

Recap

In the last session we should have:

- made a html experiment
- load in JS scripts
- used local files for JsPsych

What is JATOS

Just Another Tool for Online Studies (JATOS) is an application that allows researchers to easily host experiments on a server. It allows for pilotting, storage of data, creation of url links and lots of other things.

It is an easy way to get your experiment online.

How to use it

JATOS works through a graphical user interface, meaning you need to know where to click to make things work.

Typically this is a standard workflow:

  1. Write your experiment using a working directory, i.e. have an experiment.html file, a folder with JsPsych, stimuli or anything else you need to make the experiment run

  2. Make sure the experiment works through your browser

  3. Edit your html code so that it works with JATOS

  4. Pilot this through a local instance of JATOS

  5. Check the data looks alright

  6. Export the experiment files as a .jzip and import it to the main server

  7. Distribute links to participants

  8. Download your data

Local instance of JATOS

Follow the instructions for downloading a local instance of JATOS

https://www.jatos.org/Installation.html

Make sure the folder called jatos_mac_java or jatos_win_java is somewhere on your computer that is sensible. It will be important to find and use the folder later.

Log in

Now you should be able to access the log in page on your browser, log in using admin as the username and admin as the password.

This is a local instance, so it is not accessible to anybody other than you on your computer, it is not like accessing Google.

New study

In the top left of the page there should be a button called + new study, click on it and you see this screen

Now we need to fill in two pieces of information:

For the title option, call it demo, this will be the name of the experiment

For the Study assets' directory name also call it demo, this will be the name of the folder where all your experiment files will be

Click create

study_assets_root

Go back to the jatos_mac_java or jatos_win_java folder.

Find the sub folder called study_assets_root this is the main folder you will need to use when working with JATOS locally.

It is where your experiment directories will be.

Every time you create a new study, JATOS will create a folder based on what on you call the Study assets' directory name.

You should see a folder called demo as we created this.

There is nothing in this folder yet but we can add any experiment files needed to the folder.

From the last worksheet, we created a experiment.html file, that loaded scripts from a jspsych folder.

Add these to your demo folder in the study_assets_root.

Modifications for html files with JATOS

There are a few edits that need to be made to your experiment.html file in order to make it compatible with JATOS.

This is how our file looks now:

<!DOCTYPE html>
<html>
<head>
  <title>Page Title</title>
  <script src="jspsych/dist/jspsych.js"></script>
  <script src="jspsych/dist/plugin-html-keyboard-response.js"></script>
  <link href="jspsych/dist/jspsych.css" rel="stylesheet" type="text/css" />
</head>
<body>
</body>

<script>
  
// inititate jspsych
var jsPsych = initJsPsych();

// start timeline
timeline = [];

// define welcome message trial
var welcome = {
  type: jsPsychHtmlKeyboardResponse,
  stimulus: "Welcome to the experiment. Press any key to begin."
};

// push to timeline
timeline.push(welcome);

//run the experiment
jsPsych.run(timeline);

</script>

</html>

The following edits need to be made for it run:

1. jatos.js

We need to a script that loads in jatos.js:

<script src="jatos.js"></script>

Put this where you load in the other scripts, e.g. on a line after you load in jspsych.js

This file is not visible in the directory but it is loaded by jatos, so you do not need to see the actual file for this to work.

2. initJsPsych

We also need to edit the way jspsych is initiated.

Currently we have:

var jsPsych = initJsPsych();

We need to modify this so that JATOS can save the data after each trial:

var jsPsych = initJsPsych(
  {
    on_trial_finish: function(data) {
      var subjectID = jatos.studyResultId;
      jsPsych.data.addProperties({subject: subjectID});
      jatos.uploadResultFile(jsPsych.data.get().csv(), "Id_" + subjectID + "_trials.csv")
    },
    on_finish: function() {
      jatos.endStudy(jsPsych.data.get().csv());
    }
  }
);

The code does the following:

  • on_trial_finish: function(data) runs a function after every trial in the experiment has been finished. The following parts within the { } are what the function does

  • var subjectID = jatos.studyResultId; makes a subject ID based on a JATOS assigned value

  • jsPsych.data.addProperties({subject: subjectID}); adds the subjectID variable to the data as a column called subject

  • jatos.uploadResultFile(jsPsych.data.get().csv(), "Id_" + subjectID + "_trials.csv") uploads the data as a .csv file, the name of the file will be ID_x_trials.csv where x is the subject ID

  • on_finish: function() runs a function at the very end of the experiment

  • jatos.endStudy(jsPsych.data.get().csv()); ends the study and adds the data to JATOS

3. jsPsych.run

We also need to change the way the timeline is run. Currently we have:

jsPsych.run(timeline);

We need to change this to be compatible with JATOS:

jatos.onLoad(() => {
  jsPsych.run(timeline);
});

Final code

This is the final code. Save it as experiment.html in the demo folder.

<!DOCTYPE html>
<html>
<head>
  <title>Page Title</title>
  <script src="jspsych/dist/jspsych.js"></script>
  <script src="jatos.js"></script>
  <script src="jspsych/dist/plugin-html-keyboard-response.js"></script>
  <link href="jspsych/dist/jspsych.css" rel="stylesheet" type="text/css" />
</head>
<body>
</body>

<script>

// inititate jspsych
var jsPsych = initJsPsych(
  {
    on_trial_finish: function(data) {
      var subjectID = jatos.studyResultId;
      jsPsych.data.addProperties({subject: subjectID});
      jatos.uploadResultFile(jsPsych.data.get().csv(), "Id_" + subjectID + "_trials.csv")
    },
    on_finish: function () {
      jatos.endStudy(jsPsych.data.get().csv());
    }
  }
);

// start timeline
timeline = [];

// define welcome message trial
var welcome = {
  type: jsPsychHtmlKeyboardResponse,
  stimulus: "Welcome to the experiment. Press any key to begin."
};

// push to timeline
timeline.push(welcome);

//run the experiment
jatos.onLoad(() => {
  jsPsych.run(timeline);
});

</script>

</html>

Components

We now need to set up a component so that JATOS can load in the experiment.html file.

On the JATOS dashboard for the demo experiment there should be a light blue button called + new component, click on this and there will be the following:

Make the title called experiment

Make the HTML file path called experiment.html

Click create

You should now see the experiment component in the dashboard

Click run to test the experiment works

Data

Once you have tried the experiment, return back to the dashboard.

There will be a button called Study Results click on this to see the data:

You can download the data by clicking on the Export Results button, choosing Files only

This will give you all the .csv files as a .zip folder.

Exporting

Once you are happy with the experiment and are sure that the data are coded in a useable way, you can now export it from the local JATOS server.

Return to the demo dashboard.

Click on the Export button.

You will now have a demo.jzip file downloaded.

JATOS FF server

We will now need to host the experiment on the JATOS FF server. This is not a local server, but one that can be accessed by anybody with a login.

Go to this link:

https://jatos.ff.cuni.cz/jatos/login

Log in with the following:

Username: your surname in lowercase without diacritics, e.g. if your name is Jan Chromý, the username will be chromy

Password: your username with jatos at the end, e.g. chromyjatos

You can change the password once you log in

Import study

At the top of the dashboard there is the option to import study.

Import your demo.jzip file.

You should now see it in the dashboard.

Running and data

Everything on the FF JATOS server works the same. If you want to run the experiment to test it works, click Run.

To download the data, go to Study Results, click Export files then choose All.

LS0tCnRpdGxlOiAiVXNpbmcganNQc3ljaCBhbmQgY29nbml0aW9uLnJ1biIKc3VidGl0bGU6ICJTZXNzaW9uIDcgLSBKQVRPUyIKYXV0aG9yOiAiSmFtZXMgQnJhbmQiCmRhdGU6ICJgciBmb3JtYXQoU3lzLnRpbWUoKSwgJyVkICVCLCAlWScpYCIKb3V0cHV0OgogIHJtZGZvcm1hdHM6OnJlYWR0aGVkb3duOgogICAgcGFuZG9jX2FyZ3M6ICItLWhpZ2hsaWdodC1zdHlsZT1teS50aGVtZSIKICAgIGhpZ2hsaWdodDogcHlnbWVudHMKICAgIHRvY19mbG9hdDogdHJ1ZQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQogICAgdG9jX2RlcHRoOiAzCiAgICBjb2xsYXBzZWQ6IGZhbHNlCiAgICBkZl9wcmludDogcGFnZWQKICAgIGxpZ2h0Ym94OiBUUlVFCiAgICBnYWxsZXJ5OiBUUlVFCiAgICBjc3M6ICJjc3Mvc3R5bGUuY3NzIgogICAgCi0tLQoKYGBge3IgZWNobz1GQUxTRSwgd2FybmluZz1GQUxTRX0KbGlicmFyeShzbGlja1IpCmxpYnJhcnkoaHRtbHRvb2xzKQpsaWJyYXJ5KHhhcmluZ2FuRXh0cmEpCmxpYnJhcnkocm1hcmtkb3duKQpsaWJyYXJ5KGZvbnRhd2Vzb21lKQpsaWJyYXJ5KGJzcGx1cykKbGlicmFyeShEVCkKCmBgYAoKYGBge3Igc2V0dXAsIHdhcm5pbmc9RkFMU0UsIGVjaG89RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgIGV2YWwgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgIGNvbW1lbnQgPSBOQSwKICAgICAgICAgICAgICAgICAgICAgIG1lc3NhZ2UgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgIHdhcm5pbmcgPSBGQUxTRSkKCmtuaXRyOjprbml0X2hvb2tzJHNldCgKICBtZXNzYWdlID0gZnVuY3Rpb24oeCwgb3B0aW9ucykgewogICAgIHBhc3RlKCc8YnV0dG9uIHR5cGU9ImJ1dHRvbiIgY2xhc3M9ImNvbGxhcHNpYmxlMSI+PHN0cm9uZz4nLAogICAgIGZhKG5hbWUgPSAiY2lyY2xlLWluZm8iKSwKICAgICAnIG1vcmUgaW5mbzwvc3Ryb25nPjwvYnV0dG9uPicsICc8ZGl2IGNsYXNzPSJjb250ZW50MSI+PHA+JywKICAgICBnc3ViKCcjIycsICdcbicsIHgpLAogICAgICc8L3A+PC9kaXY+JywKICAgICBzZXAgPSAnXG4nKQogICB9KQoKY29kZWJsb2NrID0gZnVuY3Rpb24oeCwgb3B0aW9ucykgewogICAgIGNhdChwYXN0ZSgnPGRpdiBjbGFzcz0iY29kZWJsb2NrIj4nLAogICAgIHBhc3RlMCh4KSwKICAgICAnPC9kaXY+JywKICAgICBzZXAgPSAnXG4nKSkKICAgfQoKYGBgCgotLS0KCiMjIGByIGZhKCJsYW5ndWFnZSIpYCBUcmFuc2xhdGlvbnMgYXZhaWxhYmxlCgpEaXNjbGFpbWVyOiBtYXkgbm90IGJlIHZlcnkgYWNjdXJhdGUuLi4KCjxkaXYgaWQ9Imdvb2dsZV90cmFuc2xhdGVfZWxlbWVudCI+PC9kaXY+CgotLS0KCiMgV29ya3NoZWV0IG92ZXJ2aWV3CgojIyBgciBmYSgiY3Jvc3NoYWlycyIpYCBBaW1zCgpCeSB0aGUgZW5kIG9mIHRoaXMgd29ya3NoZWV0IHlvdSBzaG91bGQgYmUgYWJsZSB0bzoKCi0gKipwcm9ncmFtKiogeW91ciBvd24gZXhwZXJpbWVudHMgaW4ganNQc3ljaAotICoqaG9zdCoqIHRoZSBleHBlcmltZW50IG9ubGluZSB1c2luZyBjb2duaXRpb24ucnVuCi0gKip1c2UqKiB0aGUgcGFydGljaXBhbnQgZGF0YSBmb3IgYW5hbHlzaXMKLSAqKmFwcGx5KiogdGhlIGJhc2ljIHNraWxscyB5b3UgaGF2ZSBsZWFybnQgZm9yIHlvdXIgb3duIHB1cnBvc2VzCi0gKipsZWFybioqIHNvbWUgZXh0cmEgc2tpbGxzIHN1Y2ggYXMgSFRNTCwgamF2YXNjcmlwdCwgQ1NTIGFuZCBKU09OCgojIyBgciBmYSgidXNlci1ncmFkdWF0ZSIpYCBQcmUtcmVxdWlzaXRlcwoKVG8gY29tcGxldGUgdGhlIGFpbXMgeW91IHdpbGwgbmVlZCB0bzoKCi0gKipmb2xsb3cqKiB0aGlzIHdvcmtzaGVldAotICoqYXNrKiogcXVlc3Rpb25zIGlmIHlvdSBhcmUgbm90IHN1cmUvYmUgYWJsZSB0byBnb29nbGUKLSAqKmhhdmUqKiBhIHdvcmtpbmcgY29tcHV0ZXIgYW5kIGludGVybmV0IGNvbm5lY3Rpb24KLSAqKmJlIHBhdGllbnQqKiB3aGVuIHRoaW5ncyBkbyBub3Qgd29yawoKWW91IGRvIG5vdCBuZWVkIHRvOgoKLSBoYXZlIGFueSAqKnByb2dyYW1taW5nIGtub3dsZWRnZSoqCi0gaGF2ZSBoaWdoICoqY29tcHV0ZXIgbGl0ZXJhY3kqKgotIGtub3cgYW55dGhpbmcgYWJvdXQgKipqc1BzeWNoLCBjb2duaXRpb24ucnVuLCBodG1sLCBjc3Mgb3IgamF2YXNjcmlwdCoqCi0gYmUgYSAqKmxpbmd1aXN0KioKCiMjIGByIGZhKCJmb2xkZXItdHJlZSIpYCBTdHJ1Y3R1cmUKClRoZSB3b3Jrc2hlZXQgd2lsbCBnbyB0aHJvdWdoIHRoZSBmb2xsb3dpbmcgc2VjdGlvbnM6CgogICAgLSB3b3JraW5nIHdpdGggSkFUT1MKICAgIC0gd29ya2luZyB3aXRoIHBpbG90aW5nIG9uIGEgbG9jYWwgc2VydmVyCiAgICAtIHdvcmtpbmcgd2l0aCB0aGUgRkYgc2VydmVyCgojIyBgciBmYSgibGlnaHRidWxiIilgIFJlY2FwCgpJbiB0aGUgbGFzdCBzZXNzaW9uIHdlIHNob3VsZCBoYXZlOgoKICAgIC0gbWFkZSBhIGh0bWwgZXhwZXJpbWVudAogICAgLSBsb2FkIGluIEpTIHNjcmlwdHMKICAgIC0gdXNlZCBsb2NhbCBmaWxlcyBmb3IgSnNQc3ljaAoKLS0tCgojIFdoYXQgaXMgSkFUT1MKCkp1c3QgQW5vdGhlciBUb29sIGZvciBPbmxpbmUgU3R1ZGllcyAoSkFUT1MpIGlzIGFuIGFwcGxpY2F0aW9uIHRoYXQgYWxsb3dzIHJlc2VhcmNoZXJzIHRvIGVhc2lseSBob3N0IGV4cGVyaW1lbnRzIG9uIGEgc2VydmVyLiBJdCBhbGxvd3MgZm9yIHBpbG90dGluZywgc3RvcmFnZSBvZiBkYXRhLCBjcmVhdGlvbiBvZiB1cmwgbGlua3MgYW5kIGxvdHMgb2Ygb3RoZXIgdGhpbmdzLgoKSXQgaXMgYW4gZWFzeSB3YXkgdG8gZ2V0IHlvdXIgZXhwZXJpbWVudCBvbmxpbmUuCgojIEhvdyB0byB1c2UgaXQKCkpBVE9TIHdvcmtzIHRocm91Z2ggYSBncmFwaGljYWwgdXNlciBpbnRlcmZhY2UsIG1lYW5pbmcgeW91IG5lZWQgdG8ga25vdyB3aGVyZSB0byBjbGljayB0byBtYWtlIHRoaW5ncyB3b3JrLgoKVHlwaWNhbGx5IHRoaXMgaXMgYSBzdGFuZGFyZCB3b3JrZmxvdzoKCjEuIFdyaXRlIHlvdXIgZXhwZXJpbWVudCB1c2luZyBhIHdvcmtpbmcgZGlyZWN0b3J5LCBpLmUuIGhhdmUgYW4gYGV4cGVyaW1lbnQuaHRtbGAgZmlsZSwgYSBmb2xkZXIgd2l0aCBgSnNQc3ljaGAsIGBzdGltdWxpYCBvciBhbnl0aGluZyBlbHNlIHlvdSBuZWVkIHRvIG1ha2UgdGhlIGV4cGVyaW1lbnQgcnVuCgoyLiBNYWtlIHN1cmUgdGhlIGV4cGVyaW1lbnQgd29ya3MgdGhyb3VnaCB5b3VyIGJyb3dzZXIKCjMuIEVkaXQgeW91ciBodG1sIGNvZGUgc28gdGhhdCBpdCB3b3JrcyB3aXRoIEpBVE9TCgo0LiBQaWxvdCB0aGlzIHRocm91Z2ggYSBsb2NhbCBpbnN0YW5jZSBvZiBKQVRPUwoKNS4gQ2hlY2sgdGhlIGRhdGEgbG9va3MgYWxyaWdodAoKNi4gRXhwb3J0IHRoZSBleHBlcmltZW50IGZpbGVzIGFzIGEgLmBqemlwYCBhbmQgaW1wb3J0IGl0IHRvIHRoZSBtYWluIHNlcnZlcgoKNy4gRGlzdHJpYnV0ZSBsaW5rcyB0byBwYXJ0aWNpcGFudHMKCjguIERvd25sb2FkIHlvdXIgZGF0YQoKIyBMb2NhbCBpbnN0YW5jZSBvZiBKQVRPUwoKRm9sbG93IHRoZSBpbnN0cnVjdGlvbnMgZm9yIGRvd25sb2FkaW5nIGEgbG9jYWwgaW5zdGFuY2Ugb2YgSkFUT1MKCmh0dHBzOi8vd3d3LmphdG9zLm9yZy9JbnN0YWxsYXRpb24uaHRtbAoKTWFrZSBzdXJlIHRoZSBmb2xkZXIgY2FsbGVkIGBqYXRvc19tYWNfamF2YWAgb3IgYGphdG9zX3dpbl9qYXZhYCBpcyBzb21ld2hlcmUgb24geW91ciBjb21wdXRlciB0aGF0IGlzIHNlbnNpYmxlLiBJdCB3aWxsIGJlIGltcG9ydGFudCB0byBmaW5kIGFuZCB1c2UgdGhlIGZvbGRlciBsYXRlci4KCiMjIExvZyBpbgoKTm93IHlvdSBzaG91bGQgYmUgYWJsZSB0byBhY2Nlc3MgdGhlIGxvZyBpbiBwYWdlIG9uIHlvdXIgYnJvd3NlciwgbG9nIGluIHVzaW5nIGBhZG1pbmAgYXMgdGhlIHVzZXJuYW1lIGFuZCBgYWRtaW5gIGFzIHRoZSBwYXNzd29yZC4KClRoaXMgaXMgYSBsb2NhbCBpbnN0YW5jZSwgc28gaXQgaXMgbm90IGFjY2Vzc2libGUgdG8gYW55Ym9keSBvdGhlciB0aGFuIHlvdSBvbiB5b3VyIGNvbXB1dGVyLCBpdCBpcyBub3QgbGlrZSBhY2Nlc3NpbmcgR29vZ2xlLgoKIVtdKGltYWdlcy9qYXRvc19sb2dpbi5wbmcpCgojIyBOZXcgc3R1ZHkKCkluIHRoZSB0b3AgbGVmdCBvZiB0aGUgcGFnZSB0aGVyZSBzaG91bGQgYmUgYSBidXR0b24gY2FsbGVkIGArIG5ldyBzdHVkeWAsIGNsaWNrIG9uIGl0IGFuZCB5b3Ugc2VlIHRoaXMgc2NyZWVuCgohW10oaW1hZ2VzL2phdG9zX25ld19zdHVkeV9zZXR1cC5wbmcpCgpOb3cgd2UgbmVlZCB0byBmaWxsIGluIHR3byBwaWVjZXMgb2YgaW5mb3JtYXRpb246CgpGb3IgdGhlIGB0aXRsZWAgb3B0aW9uLCBjYWxsIGl0IGBkZW1vYCwgdGhpcyB3aWxsIGJlIHRoZSBuYW1lIG9mIHRoZSBleHBlcmltZW50CgpGb3IgdGhlIGBTdHVkeSBhc3NldHMnIGRpcmVjdG9yeSBuYW1lYCBhbHNvIGNhbGwgaXQgYGRlbW9gLCB0aGlzIHdpbGwgYmUgdGhlIG5hbWUgb2YgdGhlIGZvbGRlciB3aGVyZSBhbGwgeW91ciBleHBlcmltZW50IGZpbGVzIHdpbGwgYmUKCkNsaWNrIGBjcmVhdGVgCgojIyBzdHVkeV9hc3NldHNfcm9vdAoKR28gYmFjayB0byB0aGUgYGphdG9zX21hY19qYXZhYCBvciBgamF0b3Nfd2luX2phdmFgIGZvbGRlci4KCkZpbmQgdGhlIHN1YiBmb2xkZXIgY2FsbGVkIGBzdHVkeV9hc3NldHNfcm9vdGAgdGhpcyBpcyB0aGUgbWFpbiBmb2xkZXIgeW91IHdpbGwgbmVlZCB0byB1c2Ugd2hlbiB3b3JraW5nIHdpdGggSkFUT1MgbG9jYWxseS4KCkl0IGlzIHdoZXJlIHlvdXIgZXhwZXJpbWVudCBkaXJlY3RvcmllcyB3aWxsIGJlLgoKRXZlcnkgdGltZSB5b3UgY3JlYXRlIGEgbmV3IHN0dWR5LCBKQVRPUyB3aWxsIGNyZWF0ZSBhIGZvbGRlciBiYXNlZCBvbiB3aGF0IG9uIHlvdSBjYWxsIHRoZSBgU3R1ZHkgYXNzZXRzJyBkaXJlY3RvcnkgbmFtZWAuCgpZb3Ugc2hvdWxkIHNlZSBhIGZvbGRlciBjYWxsZWQgYGRlbW9gIGFzIHdlIGNyZWF0ZWQgdGhpcy4KCiFbXShpbWFnZXMvamF0b3NfZGlyZWN0b3J5LnBuZykKClRoZXJlIGlzIG5vdGhpbmcgaW4gdGhpcyBmb2xkZXIgeWV0IGJ1dCB3ZSBjYW4gYWRkIGFueSBleHBlcmltZW50IGZpbGVzIG5lZWRlZCB0byB0aGUgZm9sZGVyLgoKRnJvbSB0aGUgbGFzdCB3b3Jrc2hlZXQsIHdlIGNyZWF0ZWQgYSBgZXhwZXJpbWVudC5odG1sYCBmaWxlLCB0aGF0IGxvYWRlZCBzY3JpcHRzIGZyb20gYSBganNwc3ljaGAgZm9sZGVyLgoKQWRkIHRoZXNlIHRvIHlvdXIgYGRlbW9gIGZvbGRlciBpbiB0aGUgYHN0dWR5X2Fzc2V0c19yb290YC4KCiFbXShpbWFnZXMvamF0b3NfZGlyZWN0b3J5X2RlbW8ucG5nKQoKIyMgTW9kaWZpY2F0aW9ucyBmb3IgaHRtbCBmaWxlcyB3aXRoIEpBVE9TCgpUaGVyZSBhcmUgYSBmZXcgZWRpdHMgdGhhdCBuZWVkIHRvIGJlIG1hZGUgdG8geW91ciBgZXhwZXJpbWVudC5odG1sYCBmaWxlIGluIG9yZGVyIHRvIG1ha2UgaXQgY29tcGF0aWJsZSB3aXRoIEpBVE9TLgoKVGhpcyBpcyBob3cgb3VyIGZpbGUgbG9va3Mgbm93OgoKYGBge2h0bWx9CjwhRE9DVFlQRSBodG1sPgo8aHRtbD4KPGhlYWQ+CiAgPHRpdGxlPlBhZ2UgVGl0bGU8L3RpdGxlPgogIDxzY3JpcHQgc3JjPSJqc3BzeWNoL2Rpc3QvanNwc3ljaC5qcyI+PC9zY3JpcHQ+CiAgPHNjcmlwdCBzcmM9ImpzcHN5Y2gvZGlzdC9wbHVnaW4taHRtbC1rZXlib2FyZC1yZXNwb25zZS5qcyI+PC9zY3JpcHQ+CiAgPGxpbmsgaHJlZj0ianNwc3ljaC9kaXN0L2pzcHN5Y2guY3NzIiByZWw9InN0eWxlc2hlZXQiIHR5cGU9InRleHQvY3NzIiAvPgo8L2hlYWQ+Cjxib2R5Pgo8L2JvZHk+Cgo8c2NyaXB0PgogIAovLyBpbml0aXRhdGUganNwc3ljaAp2YXIganNQc3ljaCA9IGluaXRKc1BzeWNoKCk7CgovLyBzdGFydCB0aW1lbGluZQp0aW1lbGluZSA9IFtdOwoKLy8gZGVmaW5lIHdlbGNvbWUgbWVzc2FnZSB0cmlhbAp2YXIgd2VsY29tZSA9IHsKICB0eXBlOiBqc1BzeWNoSHRtbEtleWJvYXJkUmVzcG9uc2UsCiAgc3RpbXVsdXM6ICJXZWxjb21lIHRvIHRoZSBleHBlcmltZW50LiBQcmVzcyBhbnkga2V5IHRvIGJlZ2luLiIKfTsKCi8vIHB1c2ggdG8gdGltZWxpbmUKdGltZWxpbmUucHVzaCh3ZWxjb21lKTsKCi8vcnVuIHRoZSBleHBlcmltZW50CmpzUHN5Y2gucnVuKHRpbWVsaW5lKTsKCjwvc2NyaXB0PgoKPC9odG1sPgoKYGBgCgpUaGUgZm9sbG93aW5nIGVkaXRzIG5lZWQgdG8gYmUgbWFkZSBmb3IgaXQgcnVuOgoKIyMjIDEuIGphdG9zLmpzCgpXZSBuZWVkIHRvIGEgc2NyaXB0IHRoYXQgbG9hZHMgaW4gYGphdG9zLmpzYDoKCmBgYHtodG1sfQo8c2NyaXB0IHNyYz0iamF0b3MuanMiPjwvc2NyaXB0PgoKYGBgCgpQdXQgdGhpcyB3aGVyZSB5b3UgbG9hZCBpbiB0aGUgb3RoZXIgc2NyaXB0cywgZS5nLiBvbiBhIGxpbmUgYWZ0ZXIgeW91IGxvYWQgaW4gYGpzcHN5Y2guanNgCgpUaGlzIGZpbGUgaXMgbm90IHZpc2libGUgaW4gdGhlIGRpcmVjdG9yeSBidXQgaXQgaXMgbG9hZGVkIGJ5IGphdG9zLCBzbyB5b3UgZG8gbm90IG5lZWQgdG8gc2VlIHRoZSBhY3R1YWwgZmlsZSBmb3IgdGhpcyB0byB3b3JrLgoKIyMjIDIuIGluaXRKc1BzeWNoCgpXZSBhbHNvIG5lZWQgdG8gZWRpdCB0aGUgd2F5IGpzcHN5Y2ggaXMgaW5pdGlhdGVkLgoKQ3VycmVudGx5IHdlIGhhdmU6CgpgYGB7anN9CnZhciBqc1BzeWNoID0gaW5pdEpzUHN5Y2goKTsKCmBgYAoKV2UgbmVlZCB0byBtb2RpZnkgdGhpcyBzbyB0aGF0IEpBVE9TIGNhbiBzYXZlIHRoZSBkYXRhIGFmdGVyIGVhY2ggdHJpYWw6CgpgYGB7anN9CnZhciBqc1BzeWNoID0gaW5pdEpzUHN5Y2goCiAgewogICAgb25fdHJpYWxfZmluaXNoOiBmdW5jdGlvbihkYXRhKSB7CiAgICAgIHZhciBzdWJqZWN0SUQgPSBqYXRvcy5zdHVkeVJlc3VsdElkOwogICAgICBqc1BzeWNoLmRhdGEuYWRkUHJvcGVydGllcyh7c3ViamVjdDogc3ViamVjdElEfSk7CiAgICAgIGphdG9zLnVwbG9hZFJlc3VsdEZpbGUoanNQc3ljaC5kYXRhLmdldCgpLmNzdigpLCAiSWRfIiArIHN1YmplY3RJRCArICJfdHJpYWxzLmNzdiIpCiAgICB9LAogICAgb25fZmluaXNoOiBmdW5jdGlvbigpIHsKICAgICAgamF0b3MuZW5kU3R1ZHkoanNQc3ljaC5kYXRhLmdldCgpLmNzdigpKTsKICAgIH0KICB9Cik7CgpgYGAKClRoZSBjb2RlIGRvZXMgdGhlIGZvbGxvd2luZzoKCi0gYG9uX3RyaWFsX2ZpbmlzaDogZnVuY3Rpb24oZGF0YSlgIHJ1bnMgYSBmdW5jdGlvbiBhZnRlciBldmVyeSB0cmlhbCBpbiB0aGUgZXhwZXJpbWVudCBoYXMgYmVlbiBmaW5pc2hlZC4gVGhlIGZvbGxvd2luZyBwYXJ0cyB3aXRoaW4gdGhlIGB7IH1gIGFyZSB3aGF0IHRoZSBmdW5jdGlvbiBkb2VzCgotIGB2YXIgc3ViamVjdElEID0gamF0b3Muc3R1ZHlSZXN1bHRJZDtgIG1ha2VzIGEgc3ViamVjdCBJRCBiYXNlZCBvbiBhIEpBVE9TIGFzc2lnbmVkIHZhbHVlCgotIGBqc1BzeWNoLmRhdGEuYWRkUHJvcGVydGllcyh7c3ViamVjdDogc3ViamVjdElEfSk7YCBhZGRzIHRoZSBzdWJqZWN0SUQgdmFyaWFibGUgdG8gdGhlIGRhdGEgYXMgYSBjb2x1bW4gY2FsbGVkIGBzdWJqZWN0YAoKLSBgamF0b3MudXBsb2FkUmVzdWx0RmlsZShqc1BzeWNoLmRhdGEuZ2V0KCkuY3N2KCksICJJZF8iICsgc3ViamVjdElEICsgIl90cmlhbHMuY3N2IilgIHVwbG9hZHMgdGhlIGRhdGEgYXMgYSBgLmNzdmAgZmlsZSwgdGhlIG5hbWUgb2YgdGhlIGZpbGUgd2lsbCBiZSBgSURfeF90cmlhbHMuY3N2YCB3aGVyZSB4IGlzIHRoZSBzdWJqZWN0IElECgotIGBvbl9maW5pc2g6IGZ1bmN0aW9uKClgIHJ1bnMgYSBmdW5jdGlvbiBhdCB0aGUgdmVyeSBlbmQgb2YgdGhlIGV4cGVyaW1lbnQKCi0gYGphdG9zLmVuZFN0dWR5KGpzUHN5Y2guZGF0YS5nZXQoKS5jc3YoKSk7YCBlbmRzIHRoZSBzdHVkeSBhbmQgYWRkcyB0aGUgZGF0YSB0byBKQVRPUwoKIyMjIDMuIGpzUHN5Y2gucnVuCgpXZSBhbHNvIG5lZWQgdG8gY2hhbmdlIHRoZSB3YXkgdGhlIHRpbWVsaW5lIGlzIHJ1bi4gQ3VycmVudGx5IHdlIGhhdmU6CgpgYGB7anN9CmpzUHN5Y2gucnVuKHRpbWVsaW5lKTsKCmBgYAoKV2UgbmVlZCB0byBjaGFuZ2UgdGhpcyB0byBiZSBjb21wYXRpYmxlIHdpdGggSkFUT1M6CgpgYGB7anN9CmphdG9zLm9uTG9hZCgoKSA9PiB7CiAganNQc3ljaC5ydW4odGltZWxpbmUpOwp9KTsKCmBgYAoKIyMjIEZpbmFsIGNvZGUKClRoaXMgaXMgdGhlIGZpbmFsIGNvZGUuIFNhdmUgaXQgYXMgYGV4cGVyaW1lbnQuaHRtbGAgaW4gdGhlIGBkZW1vYCBmb2xkZXIuCgpgYGB7aHRtbH0KPCFET0NUWVBFIGh0bWw+CjxodG1sPgo8aGVhZD4KICA8dGl0bGU+UGFnZSBUaXRsZTwvdGl0bGU+CiAgPHNjcmlwdCBzcmM9ImpzcHN5Y2gvZGlzdC9qc3BzeWNoLmpzIj48L3NjcmlwdD4KICA8c2NyaXB0IHNyYz0iamF0b3MuanMiPjwvc2NyaXB0PgogIDxzY3JpcHQgc3JjPSJqc3BzeWNoL2Rpc3QvcGx1Z2luLWh0bWwta2V5Ym9hcmQtcmVzcG9uc2UuanMiPjwvc2NyaXB0PgogIDxsaW5rIGhyZWY9ImpzcHN5Y2gvZGlzdC9qc3BzeWNoLmNzcyIgcmVsPSJzdHlsZXNoZWV0IiB0eXBlPSJ0ZXh0L2NzcyIgLz4KPC9oZWFkPgo8Ym9keT4KPC9ib2R5PgoKPHNjcmlwdD4KCi8vIGluaXRpdGF0ZSBqc3BzeWNoCnZhciBqc1BzeWNoID0gaW5pdEpzUHN5Y2goCiAgewogICAgb25fdHJpYWxfZmluaXNoOiBmdW5jdGlvbihkYXRhKSB7CiAgICAgIHZhciBzdWJqZWN0SUQgPSBqYXRvcy5zdHVkeVJlc3VsdElkOwogICAgICBqc1BzeWNoLmRhdGEuYWRkUHJvcGVydGllcyh7c3ViamVjdDogc3ViamVjdElEfSk7CiAgICAgIGphdG9zLnVwbG9hZFJlc3VsdEZpbGUoanNQc3ljaC5kYXRhLmdldCgpLmNzdigpLCAiSWRfIiArIHN1YmplY3RJRCArICJfdHJpYWxzLmNzdiIpCiAgICB9LAogICAgb25fZmluaXNoOiBmdW5jdGlvbiAoKSB7CiAgICAgIGphdG9zLmVuZFN0dWR5KGpzUHN5Y2guZGF0YS5nZXQoKS5jc3YoKSk7CiAgICB9CiAgfQopOwoKLy8gc3RhcnQgdGltZWxpbmUKdGltZWxpbmUgPSBbXTsKCi8vIGRlZmluZSB3ZWxjb21lIG1lc3NhZ2UgdHJpYWwKdmFyIHdlbGNvbWUgPSB7CiAgdHlwZToganNQc3ljaEh0bWxLZXlib2FyZFJlc3BvbnNlLAogIHN0aW11bHVzOiAiV2VsY29tZSB0byB0aGUgZXhwZXJpbWVudC4gUHJlc3MgYW55IGtleSB0byBiZWdpbi4iCn07CgovLyBwdXNoIHRvIHRpbWVsaW5lCnRpbWVsaW5lLnB1c2god2VsY29tZSk7CgovL3J1biB0aGUgZXhwZXJpbWVudApqYXRvcy5vbkxvYWQoKCkgPT4gewogIGpzUHN5Y2gucnVuKHRpbWVsaW5lKTsKfSk7Cgo8L3NjcmlwdD4KCjwvaHRtbD4KCmBgYAoKIyMgQ29tcG9uZW50cwoKV2Ugbm93IG5lZWQgdG8gc2V0IHVwIGEgY29tcG9uZW50IHNvIHRoYXQgSkFUT1MgY2FuIGxvYWQgaW4gdGhlIGBleHBlcmltZW50Lmh0bWxgIGZpbGUuCgpPbiB0aGUgSkFUT1MgZGFzaGJvYXJkIGZvciB0aGUgZGVtbyBleHBlcmltZW50IHRoZXJlIHNob3VsZCBiZSBhIGxpZ2h0IGJsdWUgYnV0dG9uIGNhbGxlZCBgKyBuZXcgY29tcG9uZW50YCwgY2xpY2sgb24gdGhpcyBhbmQgdGhlcmUgd2lsbCBiZSB0aGUgZm9sbG93aW5nOgoKIVtdKGltYWdlcy9qYXRvc19jb21wb25lbnQucG5nKQoKTWFrZSB0aGUgYHRpdGxlYCBjYWxsZWQgYGV4cGVyaW1lbnRgCgpNYWtlIHRoZSBgSFRNTCBmaWxlIHBhdGhgIGNhbGxlZCBgZXhwZXJpbWVudC5odG1sYAoKQ2xpY2sgYGNyZWF0ZWAKCllvdSBzaG91bGQgbm93IHNlZSB0aGUgYGV4cGVyaW1lbnRgIGNvbXBvbmVudCBpbiB0aGUgZGFzaGJvYXJkCgpDbGljayBgcnVuYCB0byB0ZXN0IHRoZSBleHBlcmltZW50IHdvcmtzCgohW10oaW1hZ2VzL2phdG9zX2NvbXBvbmVudF9ydW4ucG5nKQoKIyMgRGF0YQoKT25jZSB5b3UgaGF2ZSB0cmllZCB0aGUgZXhwZXJpbWVudCwgcmV0dXJuIGJhY2sgdG8gdGhlIGRhc2hib2FyZC4KClRoZXJlIHdpbGwgYmUgYSBidXR0b24gY2FsbGVkIGBTdHVkeSBSZXN1bHRzYCBjbGljayBvbiB0aGlzIHRvIHNlZSB0aGUgZGF0YToKCiFbXShpbWFnZXMvamF0b3NfZGF0YS5wbmcpCgpZb3UgY2FuIGRvd25sb2FkIHRoZSBkYXRhIGJ5IGNsaWNraW5nIG9uIHRoZSBgRXhwb3J0IFJlc3VsdHNgIGJ1dHRvbiwgY2hvb3NpbmcgYEZpbGVzIG9ubHlgCgpUaGlzIHdpbGwgZ2l2ZSB5b3UgYWxsIHRoZSAuY3N2IGZpbGVzIGFzIGEgLnppcCBmb2xkZXIuCgojIyBFeHBvcnRpbmcKCk9uY2UgeW91IGFyZSBoYXBweSB3aXRoIHRoZSBleHBlcmltZW50IGFuZCBhcmUgc3VyZSB0aGF0IHRoZSBkYXRhIGFyZSBjb2RlZCBpbiBhIHVzZWFibGUgd2F5LCB5b3UgY2FuIG5vdyBleHBvcnQgaXQgZnJvbSB0aGUgbG9jYWwgSkFUT1Mgc2VydmVyLgoKUmV0dXJuIHRvIHRoZSBgZGVtb2AgZGFzaGJvYXJkLgoKQ2xpY2sgb24gdGhlIGBFeHBvcnRgIGJ1dHRvbi4KCllvdSB3aWxsIG5vdyBoYXZlIGEgYGRlbW8uanppcGAgZmlsZSBkb3dubG9hZGVkLgoKIyBKQVRPUyBGRiBzZXJ2ZXIKCldlIHdpbGwgbm93IG5lZWQgdG8gaG9zdCB0aGUgZXhwZXJpbWVudCBvbiB0aGUgSkFUT1MgRkYgc2VydmVyLiBUaGlzIGlzIG5vdCBhIGxvY2FsIHNlcnZlciwgYnV0IG9uZSB0aGF0IGNhbiBiZSBhY2Nlc3NlZCBieSBhbnlib2R5IHdpdGggYSBsb2dpbi4KCkdvIHRvIHRoaXMgbGluazoKCmh0dHBzOi8vamF0b3MuZmYuY3VuaS5jei9qYXRvcy9sb2dpbgoKTG9nIGluIHdpdGggdGhlIGZvbGxvd2luZzoKClVzZXJuYW1lOiB5b3VyIHN1cm5hbWUgaW4gbG93ZXJjYXNlIHdpdGhvdXQgZGlhY3JpdGljcywgZS5nLiBpZiB5b3VyIG5hbWUgaXMgSmFuIENocm9tw70sIHRoZSB1c2VybmFtZSB3aWxsIGJlIGBjaHJvbXlgCgpQYXNzd29yZDogeW91ciB1c2VybmFtZSB3aXRoIGphdG9zIGF0IHRoZSBlbmQsIGUuZy4gY2hyb215amF0b3MKCllvdSBjYW4gY2hhbmdlIHRoZSBwYXNzd29yZCBvbmNlIHlvdSBsb2cgaW4KCiMjIEltcG9ydCBzdHVkeQoKQXQgdGhlIHRvcCBvZiB0aGUgZGFzaGJvYXJkIHRoZXJlIGlzIHRoZSBvcHRpb24gdG8gYGltcG9ydCBzdHVkeWAuCgpJbXBvcnQgeW91ciBgZGVtby5qemlwYCBmaWxlLgoKWW91IHNob3VsZCBub3cgc2VlIGl0IGluIHRoZSBkYXNoYm9hcmQuCgojIyBSdW5uaW5nIGFuZCBkYXRhCgpFdmVyeXRoaW5nIG9uIHRoZSBGRiBKQVRPUyBzZXJ2ZXIgd29ya3MgdGhlIHNhbWUuIElmIHlvdSB3YW50IHRvIHJ1biB0aGUgZXhwZXJpbWVudCB0byB0ZXN0IGl0IHdvcmtzLCBjbGljayBgUnVuYC4KClRvIGRvd25sb2FkIHRoZSBkYXRhLCBnbyB0byBgU3R1ZHkgUmVzdWx0c2AsIGNsaWNrIGBFeHBvcnQgZmlsZXNgIHRoZW4gY2hvb3NlIGBBbGxgLgoKIyMgTGlua3MKClRvIGdldCBhIGxpbmsgZm9yIHlvdXIgZXhwZXJpbWVudCB0byBzZW5kIG91dCB0byBwYXJ0aWNpcGFudHM6CgoxLiBDbGljayBvbiB0aGUgZ3JlZW4gYFN0dWR5IExpbmtzYCBidXR0b24KCjIuIFNlbGVjdCBgR2VuZXJhbCBNdWx0aXBsZWAKCjMuIENsaWNrIGBTdHVkeSBMaW5rYAoKNC4gQ29weSB0aGUgbGluayBkaXNwbGF5ZWQgaW4gdGhlIGJveAoKIVtdKGltYWdlcy9qYXRvc19saW5rLnBuZykKCgpgYGB7ciBlY2hvPUZBTFNFLCBldmFsPVRSVUUsIHdhcm5pbmc9RkFMU0V9Cmh0bWx0b29sczo6dGFncyRzY3JpcHQoc3JjID0gImpzL3RyYW5zbGF0ZS5qcyIpCiMgaHRtbHRvb2xzOjp0YWdzJHNjcmlwdChzcmMgPSAianMvaW5mb2JveC5qcyIpCmh0bWx0b29sczo6dGFncyRzY3JpcHQoc3JjPSIvL3RyYW5zbGF0ZS5nb29nbGUuY29tL3RyYW5zbGF0ZV9hL2VsZW1lbnQuanM/Y2I9Z29vZ2xlVHJhbnNsYXRlRWxlbWVudEluaXQiKQoKaHRtbHRvb2xzOjp0YWdMaXN0KAogIHhhcmluZ2FuRXh0cmE6OnVzZV9jbGlwYm9hcmQoCiAgICBidXR0b25fdGV4dCA9ICI8aSBjbGFzcz1cImZhIGZhLWNsaXBib2FyZFwiIHN0eWxlPVwiZm9udC1zaXplOiAyNXB4XCI+PC9pPiIsCiAgICBzdWNjZXNzX3RleHQgPSAiPGkgY2xhc3M9XCJmYSBmYS1jaGVja1wiIHN0eWxlPVwiY29sb3I6ICM5MEJFNkQ7IGZvbnQtc2l6ZTogMjVweFwiPjwvaT4iLAogICksCiAgcm1hcmtkb3duOjpodG1sX2RlcGVuZGVuY3lfZm9udF9hd2Vzb21lKCkKKQoKYGBgCgpgYGB7anMgZWNobz1GQUxTRSwgZXZhbD1UUlVFfQp2YXIgY29sbCA9IGRvY3VtZW50LmdldEVsZW1lbnRzQnlDbGFzc05hbWUoImNvbGxhcHNpYmxlMSIpOwp2YXIgaTsKCmZvciAoaSA9IDA7IGkgPCBjb2xsLmxlbmd0aDsgaSsrKSB7CiAgY29sbFtpXS5hZGRFdmVudExpc3RlbmVyKCJjbGljayIsIGZ1bmN0aW9uKCkgewogICAgdGhpcy5jbGFzc0xpc3QudG9nZ2xlKCJhY3RpdmUxIik7CiAgICB2YXIgY29udGVudCA9IHRoaXMubmV4dEVsZW1lbnRTaWJsaW5nOwogICAgaWYgKGNvbnRlbnQuc3R5bGUubWF4SGVpZ2h0KXsKICAgICAgY29udGVudC5zdHlsZS5tYXhIZWlnaHQgPSBudWxsOwogICAgfSBlbHNlIHsKICAgICAgY29udGVudC5zdHlsZS5tYXhIZWlnaHQgPSBjb250ZW50LnNjcm9sbEhlaWdodCArICJweCI7CiAgICB9CiAgfSk7Cn0KCmBgYAo=